Deciphering the "Art" in Modeling and Simulation of the Knee Joint: Assessing Model Calibration Workflows and Outcomes.

Model reproducibility is a point of emphasis for the National Institutes of Health (NIH) and in science, broadly. As the use of computational modeling in biomechanics and orthopedics grows, so does the need to assess the reproducibility of modeling workflows and simulation predictions. The long-term goal of the KneeHub project is to understand the influence of potentially subjective decisions, thus the modeler's "art", on the reproducibility and predictive uncertainty of computational knee joint models. In this paper, we report on the model calibration phase of this project, during which five teams calibrated computational knee joint models of the same specimens from the same specimen-specific joint mechanics dataset. We investigated model calibration approaches and decisions, and compared calibration workflows and model outcomes among the teams. The selection of the calibration targets used in the calibration workflow differed greatly between the teams and was influenced by modeling decisions related to the representation of structures, and considerations for computational cost and implementation of optimization. While calibration improved model performance, differences in the postcalibration ligament properties and predicted kinematics were quantified and discussed in the context of modeling decisions. Even for teams with demonstrated expertise, model calibration is difficult to foresee and plan in detail, and the results of this study underscore the importance of identification and standardization of best practices for data sharing and calibration.

Reproducibility in modeling and simulation of the knee: Academic, industry, and regulatory perspectives.

Stakeholders in the modeling and simulation (M&S) community organized a workshop at the 2019 Annual Meeting of the Orthopaedic Research Society (ORS) entitled "Reproducibility in Modeling and Simulation of the Knee: Academic, Industry, and Regulatory Perspectives." The goal was to discuss efforts among these stakeholders to address irreproducibility in M&S focusing on the knee joint. An academic representative from a leading orthopedic hospital in the United States described a multi-institutional, open effort funded by the National Institutes of Health to assess model reproducibility in computational knee biomechanics. A regulatory representative from the United States Food and Drug Administration indicated the necessity of standards for reproducibility to increase utility of M&S in the regulatory setting. An industry representative from a major orthopedic implant company emphasized improving reproducibility by addressing indeterminacy in personalized modeling through sensitivity analyses, thereby enhancing preclinical evaluation of joint replacement technology. Thought leaders in the M&S community stressed the importance of data sharing to minimize duplication of efforts. A survey comprised 103 attendees revealed strong support for the workshop and for increasing emphasis on computational modeling at future ORS meetings. Nearly all survey respondents (97%) considered reproducibility to be an important issue. Almost half of respondents (45%) tried and failed to reproduce the work of others. Two-thirds of respondents (67%) declared that individual laboratories are most responsible for ensuring reproducible research whereas 44% thought that journals are most responsible. Thought leaders and survey respondents emphasized that computational models must be reproducible and credible to advance knee M&S.

Assessment of reporting practices and reproducibility potential of a cohort of published studies in computational knee biomechanics.

Reproducible research serves as a pillar of the scientific method and is a foundation for scientific advancement. However, estimates for irreproducibility of preclinical science range from 75% to 90%. The importance of reproducible science has not been assessed in the context of mechanics-based modeling of human joints such as the knee, despite this being an area that has seen dramatic growth. Framed in the context of five experienced teams currently documenting knee modeling procedures, the aim of this study was to evaluate reporting and the perceived potential for reproducibility across studies the teams viewed as important contributions to the literature. A cohort of studies was selected by polling, which resulted in an assessment of nine studies as opposed to a broader analysis across the literature. Using a published checklist for reporting of modeling features, the cohort was evaluated for both "reporting" and their potential to be "reproduced," which was delineated into six major modeling categories and three subcategories. Logistic regression analysis revealed that for individual modeling categories, the proportion of "reported" occurrences ranged from 0.31, 95% confidence interval (CI) [0.23, 0.41] to 0.77, 95% CI: [0.68, 0.86]. The proportion of whether a category was perceived as "reproducible" ranged from 0.22, 95% CI: [0.15, 0.31] to 0.44, 95% CI: [0.35, 0.55]. The relatively low ratios highlight an opportunity to improve reporting and reproducibility of knee modeling studies. Ongoing efforts, including our findings, contribute to a dialogue that facilitates adoption of practices that provide both credibility and translation possibilities.

A Parameter Sensitivity Analysis on Multiple Finite Element Knee Joint Models.

The reproducibility of computational knee joint modeling is questionable, with models varying depending on the modeling team. The influence of model variations on simulation outcomes should be investigated, since knowing the sensitivity of the model outcomes to model parameters could help determine which parameters to calibrate and which parameters could potentially be standardized, improving model reproducibility. Previous sensitivity analyses on finite element knee joint models have typically used one model, with a few parameters and ligaments represented as line segments. In this study, a parameter sensitivity analysis was performed using multiple finite element knee joint models with continuum ligament representations. Four previously developed and calibrated models of the tibiofemoral joint were used. Parameters of the ligament and meniscus material models, the cartilage contact formulation, the simulation control and the rigid cylindrical joints were studied. Varus-valgus simulations were performed, changing one parameter at a time. The sensitivity on model convergence, valgus kinematics, articulating cartilage contact pressure and contact pressure location were investigated. A scoring system was defined to categorize the parameters as having a "large," "medium" or "small" influence on model output. Model outcomes were sensitive to the ligament prestretch factor, Young's modulus and attachment condition parameters. Changes in the meniscus horn stiffness had a "small" influence. Of the cartilage contact parameters, the penalty factor and Augmented Lagrangian setting had a "large" influence on the cartilage contact pressure. In the rigid cylindrical joint, the largest influence on the outcome parameters was found by the moment penalty parameter, which caused convergence issues. The force penalty and gap tolerance had a "small" influence at most. For the majority of parameters, the sensitivity was model-dependent. For example, only two models showed convergence issues when changing the Quasi-Newton update method. Due to the sensitivity of the model parameters being model-specific, the sensitivity of the parameters found in one model cannot be assumed to be the same in other models. The sensitivity of the model outcomes to ligament material properties confirms that calibration of these parameters is critical and using literature values may not be appropriate.

A Method to Compare Heterogeneous Types of Bone and Cartilage Meshes.

Accurately capturing the bone and cartilage morphology and generating a mesh remains a critical step in the workflow of computational knee joint modeling. Currently, there is no standardized method to compare meshes of different element types and nodal densities, making comparisons across research teams a significant challenge. The aim of this paper is to describe a method to quantify differences in knee joint bone and cartilages meshes, independent of bone and cartilage mesh topology. Bone mesh-to-mesh distances, subchondral bone boundaries, and cartilage thicknesses from meshes of any type of mesh are obtained using a series of steps involving registration, resampling, and radial basis function fitting after which the comparisons are performed. Subchondral bone boundaries and cartilage thicknesses are calculated and visualized in a common frame of reference for comparison. The established method is applied to models developed by five modeling teams. Our approach to obtain bone mesh-to-mesh distances decreased the divergence seen in selecting a reference mesh (i.e., comparing mesh A-to-B versus mesh B-to-A). In general, the bone morphology was similar across teams. The cartilage thicknesses for all models were calculated and the mean absolute cartilage thickness difference was presented, the articulating areas had the best agreement across teams. The teams showed disagreement on the subchondral bone boundaries. The method presented in this paper allows for objective comparisons of bone and cartilage geometry that is agnostic to mesh type and nodal density.

Deciphering the "Art" in Modeling and Simulation of the Knee Joint: Variations in Model Development.

The use of computational modeling to investigate knee joint biomechanics has increased exponentially over the last few decades. Developing computational models is a creative process where decisions have to be made, subject to the modelers' knowledge and previous experiences, resulting in the "art" of modeling. The long-term goal of the KneeHub project is to understand the influence of subjective decisions on the final outcomes and the reproducibility of computational knee joint models. In this paper, we report on the model development phase of this project, investigating model development decisions and deviations from initial modeling plans. Five teams developed computational knee joint models from the same dataset, and we compared each teams' initial uncalibrated models and their model development workflows. Variations in the software tools and modeling approaches were found, resulting in differences such as the representation of the anatomical knee joint structures in the model. The teams consistently defined the boundary conditions and used the same anatomical coordinate system convention. However, deviations in the anatomical landmarks used to define the coordinate systems were present, resulting in a large spread in the kinematic outputs of the uncalibrated models. The reported differences and similarities in model development and simulation presented here illustrate the importance of the "art" of modeling and how subjective decision-making can lead to variation in model outputs. All teams deviated from their initial modeling plans, indicating that model development is a flexible process and difficult to plan in advance, even for experienced teams.